<!-- HTML header for doxygen 1.8.6-->
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
<head>
<meta http-equiv="Content-Type" content="text/xhtml;charset=UTF-8"/>
<meta http-equiv="X-UA-Compatible" content="IE=9"/>
<meta name="generator" content="Doxygen 1.8.13"/>
<title>OpenCV: High-level design overview</title>
<link href="../../opencv.ico" rel="shortcut icon" type="image/x-icon" />
<link href="../../tabs.css" rel="stylesheet" type="text/css"/>
<script type="text/javascript" src="../../jquery.js"></script>
<script type="text/javascript" src="../../dynsections.js"></script>
<script type="text/javascript" src="../../tutorial-utils.js"></script>
<link href="../../search/search.css" rel="stylesheet" type="text/css"/>
<script type="text/javascript" src="../../search/searchdata.js"></script>
<script type="text/javascript" src="../../search/search.js"></script>
<script type="text/x-mathjax-config">
  MathJax.Hub.Config({
    extensions: ["tex2jax.js", "TeX/AMSmath.js", "TeX/AMSsymbols.js"],
    jax: ["input/TeX","output/HTML-CSS"],
});
//<![CDATA[
MathJax.Hub.Config(
{
  TeX: {
      Macros: {
          matTT: [ "\\[ \\left|\\begin{array}{ccc} #1 & #2 & #3\\\\ #4 & #5 & #6\\\\ #7 & #8 & #9 \\end{array}\\right| \\]", 9],
          fork: ["\\left\\{ \\begin{array}{l l} #1 & \\mbox{#2}\\\\ #3 & \\mbox{#4}\\\\ \\end{array} \\right.", 4],
          forkthree: ["\\left\\{ \\begin{array}{l l} #1 & \\mbox{#2}\\\\ #3 & \\mbox{#4}\\\\ #5 & \\mbox{#6}\\\\ \\end{array} \\right.", 6],
          forkfour: ["\\left\\{ \\begin{array}{l l} #1 & \\mbox{#2}\\\\ #3 & \\mbox{#4}\\\\ #5 & \\mbox{#6}\\\\ #7 & \\mbox{#8}\\\\ \\end{array} \\right.", 8],
          vecthree: ["\\begin{bmatrix} #1\\\\ #2\\\\ #3 \\end{bmatrix}", 3],
          vecthreethree: ["\\begin{bmatrix} #1 & #2 & #3\\\\ #4 & #5 & #6\\\\ #7 & #8 & #9 \\end{bmatrix}", 9],
          cameramatrix: ["#1 = \\begin{bmatrix} f_x & 0 & c_x\\\\ 0 & f_y & c_y\\\\ 0 & 0 & 1 \\end{bmatrix}", 1],
          distcoeffs: ["(k_1, k_2, p_1, p_2[, k_3[, k_4, k_5, k_6 [, s_1, s_2, s_3, s_4[, \\tau_x, \\tau_y]]]]) \\text{ of 4, 5, 8, 12 or 14 elements}"],
          distcoeffsfisheye: ["(k_1, k_2, k_3, k_4)"],
          hdotsfor: ["\\dots", 1],
          mathbbm: ["\\mathbb{#1}", 1],
          bordermatrix: ["\\matrix{#1}", 1]
      }
  }
}
);
//]]>
</script><script type="text/javascript" src="https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.0/MathJax.js"></script>
<link href="../../doxygen.css" rel="stylesheet" type="text/css" />
<link href="../../stylesheet.css" rel="stylesheet" type="text/css"/>
</head>
<body>
<div id="top"><!-- do not remove this div, it is closed by doxygen! -->
<div id="titlearea">
<!--#include virtual="/google-search.html"-->
<table cellspacing="0" cellpadding="0">
 <tbody>
 <tr style="height: 56px;">
  <td id="projectlogo"><img alt="Logo" src="../../opencv-logo-small.png"/></td>
  <td style="padding-left: 0.5em;">
   <div id="projectname">OpenCV
   &#160;<span id="projectnumber">4.5.2</span>
   </div>
   <div id="projectbrief">Open Source Computer Vision</div>
  </td>
 </tr>
 </tbody>
</table>
</div>
<!-- end header part -->
<!-- Generated by Doxygen 1.8.13 -->
<script type="text/javascript">
var searchBox = new SearchBox("searchBox", "../../search",false,'Search');
</script>
<script type="text/javascript" src="../../menudata.js"></script>
<script type="text/javascript" src="../../menu.js"></script>
<script type="text/javascript">
$(function() {
  initMenu('../../',true,false,'search.php','Search');
  $(document).ready(function() { init_search(); });
});
</script>
<div id="main-nav"></div>
<!-- window showing the filter options -->
<div id="MSearchSelectWindow"
     onmouseover="return searchBox.OnSearchSelectShow()"
     onmouseout="return searchBox.OnSearchSelectHide()"
     onkeydown="return searchBox.OnSearchSelectKey(event)">
</div>

<!-- iframe showing the search results (closed by default) -->
<div id="MSearchResultsWindow">
<iframe src="javascript:void(0)" frameborder="0" 
        name="MSearchResults" id="MSearchResults">
</iframe>
</div>

<div id="nav-path" class="navpath">
  <ul>
<li class="navelem"><a class="el" href="../../d0/d1e/gapi.html">Graph API</a></li>  </ul>
</div>
</div><!-- top -->
<div class="header">
  <div class="headertitle">
<div class="title">High-level design overview </div>  </div>
</div><!--header-->
<div class="contents">
<div class="toc"><h3>Table of Contents</h3>
<ul><li class="level1"><a href="#gapi_api_layer">API layer</a></li>
<li class="level1"><a href="#gapi_compiler">Graph compiler layer</a></li>
<li class="level1"><a href="#gapi_backends">Backends layer</a></li>
<li class="level1"><a href="#gapi_compiled">Graph execution</a></li>
</ul>
</div>
<div class="textblock"><h1>G-API High-level design overview</h1>
<p>G-API is a heterogeneous framework and provides an unified API to program image processing pipelines with a number of supported backends.</p>
<p>The key design idea is to keep pipeline code itself platform-neutral while specifying which kernels to use and which devices to utilize using extra parameters at graph compile (configuration) time. This requirement has led to the following architecture:</p>
<div class="image">
<img src="../../gapi_scheme.png" alt="gapi_scheme.png"/>
<div class="caption">
G-API framework architecture</div></div>
<p> There are three layers in this architecture:</p><ul>
<li><b>API Layer</b> &ndash; this is the top layer, which implements G-API public interface, its building blocks and semantics. When user constructs a pipeline with G-API, he interacts with this layer directly, and the entities the user operates on (like <a class="el" href="../../df/daa/classcv_1_1GMat.html">cv::GMat</a> or <a class="el" href="../../d9/dfe/classcv_1_1GComputation.html" title="GComputation class represents a captured computation graph. GComputation objects form boundaries for ...">cv::GComputation</a>) are provided by this layer.</li>
<li><b>Graph Compiler Layer</b> &ndash; this is the intermediate layer which unrolls user computation into a graph and then applies a number of transformations to it (e.g. optimizations). This layer is built atop of <a class="el" href="../../d4/ddd/gapi_impl.html#gapi_detail_ade">ADE Framework</a>.</li>
<li><b>Backends Layer</b> &ndash; this is the lowest level layer, which lists a number of <em>Backends</em>. In contrast with the above two layers, backends are highly coupled with low-level platform details, with every backend standing for every platform. A backend operates on a processed graph (coming from the graph compiler) and executes this graph optimally for a specific platform or device.</li>
</ul>
<h1><a class="anchor" id="gapi_api_layer"></a>
API layer</h1>
<p>API layer is what user interacts with when defining and using a pipeline (a Computation in G-API terms). API layer defines a set of G-API <em>dynamic</em> objects which can be used as inputs, outputs, and intermediate data objects within a graph:</p><ul>
<li><a class="el" href="../../df/daa/classcv_1_1GMat.html">cv::GMat</a></li>
<li><a class="el" href="../../d9/d98/classcv_1_1GScalar.html">cv::GScalar</a></li>
<li><a class="el" href="../../d3/d44/classcv_1_1GArray.html">cv::GArray</a> (template class)</li>
</ul>
<p>API layer specifies a list of Operations which are defined on these data objects &ndash; so called kernels. See G-API <a class="el" href="../../df/d1f/group__gapi__core.html">core</a> and <a class="el" href="../../d2/d00/group__gapi__imgproc.html">imgproc</a> namespaces for details on which operations G-API provides by default.</p>
<p>G-API is not limited to these operations only &ndash; users can define their own kernels easily using a special macro <a class="el" href="../../de/d7f/gkernel_8hpp.html#a83d2e73c619a545f69e89bf7b03cbd80">G_TYPED_KERNEL()</a>.</p>
<p>API layer is also responsible for marshalling and storing operation parameters on pipeline creation. In addition to the aforementioned G-API dynamic objects, operations may also accept arbitrary parameters (more on this <a class="el" href="../../d4/ddd/gapi_impl.html#gapi_detail_params">here</a>), so API layer captures its values and stores internally upon the moment of execution.</p>
<p>Finally, <a class="el" href="../../d9/dfe/classcv_1_1GComputation.html" title="GComputation class represents a captured computation graph. GComputation objects form boundaries for ...">cv::GComputation</a> and <a class="el" href="../../d2/d2c/classcv_1_1GCompiled.html" title="Represents a compiled computation (graph). Can only be used with image / data formats &amp; resolutions i...">cv::GCompiled</a> are the remaining important components of API layer. The former wraps a series of G-API expressions into an object (graph), and the latter is a product of graph <em>compilation</em> (see <a class="el" href="../../d4/ddd/gapi_impl.html#gapi_detail_compiler">this chapter</a> for details).</p>
<h1><a class="anchor" id="gapi_compiler"></a>
Graph compiler layer</h1>
<p>Every G-API computation is compiled before it executes. Compilation process is triggered in two ways:</p><ul>
<li><em>implicitly</em>, when <a class="el" href="../../d9/dfe/classcv_1_1GComputation.html#aa25bc73f7cba9bb2f9992edaa248f868" title="Compile graph on-the-fly and immediately execute it on the inputs data vectors. ">cv::GComputation::apply()</a> is used. In this case, graph compilation is then immediately followed by execution.</li>
<li><em>explicitly</em>, when <a class="el" href="../../d9/dfe/classcv_1_1GComputation.html#ab11bcc0578f9aa5ec8f8eaf6cda0dfa8" title="Compile the computation for specific input format(s). ">cv::GComputation::compile()</a> is used. In this case, a <a class="el" href="../../d2/d2c/classcv_1_1GCompiled.html" title="Represents a compiled computation (graph). Can only be used with image / data formats &amp; resolutions i...">cv::GCompiled</a> object is returned which then can be invoked as a C++ functor.</li>
</ul>
<p>The first way is recommended for cases when input data format is not known in advance &ndash; e.g. when it comes from an arbitrary input file. The second way is recommended for deployment (production) scenarios where input data characteristics are usually predefined.</p>
<p>Graph compilation process is built atop of ADE Framework. Initially, a bipartite graph is generated from expressions captured by API layer. This graph contains nodes of two types: <em>Data</em> and <em>Operations</em>. Graph always starts and ends with a Data node(s), with Operations nodes in-between. Every Operation node has inputs and outputs, both are Data nodes.</p>
<p>After the initial graph is generated, it is actually processed by a number of graph transformations, called <em>passes</em>. ADE Framework acts as a compiler pass management engine, and passes are written specifically for G-API.</p>
<p>There are different passes which check graph validity, refine details on operations and data, organize nodes into clusters ("Islands") based on affinity or user-specified regioning[TBD], and more. Backends also are able to inject backend-specific passes into the compilation process, see more on this in the <a class="el" href="../../d4/ddd/gapi_impl.html#gapi_detail_meta">dedicated chapter</a>.</p>
<p>Result of graph compilation is a compiled object, represented by class <a class="el" href="../../d2/d2c/classcv_1_1GCompiled.html" title="Represents a compiled computation (graph). Can only be used with image / data formats &amp; resolutions i...">cv::GCompiled</a>. A new <a class="el" href="../../d2/d2c/classcv_1_1GCompiled.html" title="Represents a compiled computation (graph). Can only be used with image / data formats &amp; resolutions i...">cv::GCompiled</a> object is always created regardless if there was an explicit or implicit compilation request (see above). Actual graph execution happens within <a class="el" href="../../d2/d2c/classcv_1_1GCompiled.html" title="Represents a compiled computation (graph). Can only be used with image / data formats &amp; resolutions i...">cv::GCompiled</a> and is determined by backends which participated in the graph compilation.</p>
<dl class="section see"><dt>See also</dt><dd><a class="el" href="../../d9/dfe/classcv_1_1GComputation.html#aa25bc73f7cba9bb2f9992edaa248f868" title="Compile graph on-the-fly and immediately execute it on the inputs data vectors. ">cv::GComputation::apply()</a>, <a class="el" href="../../d9/dfe/classcv_1_1GComputation.html#ab11bcc0578f9aa5ec8f8eaf6cda0dfa8" title="Compile the computation for specific input format(s). ">cv::GComputation::compile()</a>, <a class="el" href="../../d2/d2c/classcv_1_1GCompiled.html" title="Represents a compiled computation (graph). Can only be used with image / data formats &amp; resolutions i...">cv::GCompiled</a></dd></dl>
<h1><a class="anchor" id="gapi_backends"></a>
Backends layer</h1>
<p>The above diagram lists two backends, <em>OpenCV</em> and <em>Fluid</em>. <em>OpenCV</em> is so-called "reference backend", which implements G-API operations using plain old OpenCV functions. This backend is useful for prototyping on a familiar development system. <em>Fluid</em> is a plugin for cache-efficient execution on CPU &ndash; it implements a different execution policy and operates with its own, special kernels. Fluid backend allows to achieve less memory footprint and better memory locality when running on CPU.</p>
<p>There may be more backends available, e.g. Halide, OpenCL, etc. &ndash; G-API provides an uniform internal API to develop backends so any enthusiast or a company are free to scale G-API on a new platform or accelerator. In terms of OpenCV infrastructure, every new backend is a new distinct OpenCV module, which extends G-API when build as a part of OpenCV.</p>
<h1><a class="anchor" id="gapi_compiled"></a>
Graph execution</h1>
<p>The way graph executed is defined by backends selected for compilation. In fact, every backend builds its own execution script as the final stage of graph compilation process, when an executable (compiled) object is being generated. For example, in OpenCV backend, this script is just a topologically-sorted sequence of OpenCV functions to call; for Fluid backend, it is a similar thing &ndash; a topologically sorted list of <em>Agents</em> processing lines of input on every iteration.</p>
<p>Graph execution is triggered in two ways:</p><ul>
<li>via <a class="el" href="../../d9/dfe/classcv_1_1GComputation.html#aa25bc73f7cba9bb2f9992edaa248f868" title="Compile graph on-the-fly and immediately execute it on the inputs data vectors. ">cv::GComputation::apply()</a>, with graph compiled in-place exactly for the given input data;</li>
<li>via <a class="el" href="../../d2/d2c/classcv_1_1GCompiled.html#ad2ed48a66be94fae14bc3a2a08ea8e95" title="Run the compiled computation, a generic version. ">cv::GCompiled::operator()()</a>, when the graph has been precompiled.</li>
</ul>
<p>Both methods are polimorphic and take a variadic number of arguments, with validity checks performed in runtime. If a number, shapes, and formats of passed data objects differ from expected, a run-time exception is thrown. G-API also provides <em>typed</em> wrappers to move these checks to the compile time &ndash; see <code><a class="el" href="../../dc/d07/classcv_1_1GComputationT.html" title="This class is a typed wrapper over a regular GComputation. ">cv::GComputationT</a>&lt;&gt;</code>.</p>
<p>G-API graph execution is declared stateless &ndash; it means that a compiled functor (<a class="el" href="../../d2/d2c/classcv_1_1GCompiled.html" title="Represents a compiled computation (graph). Can only be used with image / data formats &amp; resolutions i...">cv::GCompiled</a>) acts like a pure C++ function and provides the same result for the same set of input arguments.</p>
<p>Both execution methods take \(N+M\) parameters, where \(N\) is a number of inputs, and \(M\) is a number of outputs on which a <a class="el" href="../../d9/dfe/classcv_1_1GComputation.html" title="GComputation class represents a captured computation graph. GComputation objects form boundaries for ...">cv::GComputation</a> is defined. Note that while G-API types (<a class="el" href="../../df/daa/classcv_1_1GMat.html">cv::GMat</a>, etc) are used in definition, the execution methods accept OpenCV's traditional data types (like <a class="el" href="../../d3/d63/classcv_1_1Mat.html" title="n-dimensional dense array class ">cv::Mat</a>) which hold actual data &ndash; see table in <a class="el" href="../../d4/ddd/gapi_impl.html#gapi_detail_params">parameter marshalling</a>.</p>
<dl class="section see"><dt>See also</dt><dd><a class="el" href="../../d4/ddd/gapi_impl.html">Implementation details</a>, <a class="el" href="../../d0/d25/gapi_kernel_api.html">Kernel API</a> </dd></dl>
</div></div><!-- contents -->
<!-- HTML footer for doxygen 1.8.6-->
<!-- start footer part -->
<hr class="footer"/><address class="footer"><small>
Generated on Fri Apr 2 2021 11:36:33 for OpenCV by &#160;<a href="http://www.doxygen.org/index.html">
<img class="footer" src="../../doxygen.png" alt="doxygen"/>
</a> 1.8.13
</small></address>
<script type="text/javascript">
//<![CDATA[
addTutorialsButtons();
//]]>
</script>
</body>
</html>
